Skip to content

Freeze paired review experiment inputs - #18

Closed
mchwang wants to merge 15 commits into
mainfrom
codex/freeze-review-experiment-pairs
Closed

mchwang wants to merge 15 commits into
mainfrom
codex/freeze-review-experiment-pairs

Conversation

@mchwang

@mchwang mchwang commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Issue #3's software prerequisites and review inputs are complete. The human gate has not started.

This PR:

  • records Max Hwang as the blinded reviewer and Codex as the experiment operator;
  • freezes four comparable Guardyx issue pairs across documentation, agent-key security, asynchronous notifications, and compliance evidence;
  • preserves the approved alternating raw-diff/codeboost method pattern after randomizing A/B issue placement;
  • adds an explicit non-disclosure boundary for plant inputs and sealed data;
  • records the required manual assignment against issue Validate review-summary edge cases before the real-issue experiment #10 / PR Validate review-summary edge cases #17: 12 of 12 segments assigned, attributed, and in scope; and
  • records the draft PR and exact base/head revision for all eight experimental packages.

No timed paired review has started. All four pairs are prepared under the frozen rules, every exact review revision passed the Guardyx make preflight gate, no reviewer decision exists, and this PR does not claim that the gate passed. An automated review mistakenly requested on the original pair 3 and pair 4 drafts invalidated those four blinded packages; they were closed and replaced with freshly randomized packages without changing the frozen issues or method assignments. Automated review remains off experimental drafts until their human decisions are recorded.

The initial #554/#556 pair was rejected before implementation because its mandatory live telemetry baseline is unavailable from the operator environment. Inspect-before-build then caught that a proposed replacement, #335, is already fixed on current main despite remaining open. A current-tree audit excluded other stale issues and confirmed that all eight final selections remained unimplemented before work began; A/B placement was randomized only after that audit.

Protocol validation on head f4618ceb2f68af95ec1ed7e302eed24621ca7d92: git diff --check passed and every authoritative preparation-status reference agrees with the replacement table. The implementation tests previously passed with npm run typecheck and npm test -- --run test/review.test.ts test/plant.test.ts (14 tests). The manual assignment was rerun through a fresh SQLite review store from detached head 442147d, and the committed manifest matches all 12 observed stored choices with no duplicate ID.

Review-lesson audit

  • Round 1 on 8a226ea: no findings.
  • Round 2: accepted the contradictory manual-assignment status and aggregate-only assignment evidence findings; synchronized status documents and added a per-segment manifest plus a reusable evidence rule.
  • Round 3 on 576b973: no findings.
  • Round 4 on b0afccd: accepted both findings; recorded full 40-character revisions and clarified preparation status.
  • Round 5 on c8202bc: no findings.
  • Later preparation rounds repeatedly surfaced the same stale pair-status wording as new pair rows were filled. The final update now derives every authoritative status statement from the completed eight-package table and marks all four pairs prepared.
  • A final summary-only concern reproduced one stale statement in docs/implementation/build-step-1.md; it now records the completed manual assignment and the prepared but unperformed timed experiment.
  • No finding was declined.

Progresses #3.

Copilot AI lite review requested due to automatic review settings September 24, 2026 00:56

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Clarify revision evidence, preserve reviewer blinding, and resolve contradictory status text.

Review effort: Lite
Findings: None

What changed in this PR

Freezes the paired-review experiment inputs and records the completed manual attribution exercise.

Changes:

  • Defines reviewer/operator roles, issue pairs, method assignments, and disclosure rules.
  • Records 12-segment attribution results for issue #10 / PR #17.
File Description
docs/​experiments/​review-protocol.md Frozen experiment protocol, assignments, and safeguards
docs/​experiments/​manual-assignment.md Manual attribution evidence and SHA references

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot AI review requested due to automatic review settings September 24, 2026 00:58

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Resolve the outstanding documentation and evidence-traceability comments before approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 Low severity

Open (1)
Previously missed (1)

In code that hasn't changed since last review

Low severity Add per-segment assignment manifest for attribution verification

docs/​experiments/​manual-assignment.md:17

The protocol requires recording how every change maps to the attribution table, but this is only an aggregate count. It does not identify the 12 segments (for example by stable path/operation/line or stored-choice ID), so a future run cannot verify that each segment was assigned exactly once or distinguish the two P3 segments. Commit a per-segment assignment manifest or reproducible fixture output, without including plant/sealed data.

Comment thread docs/experiments/review-protocol.md
Copilot AI review requested due to automatic review settings September 24, 2026 01:00

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Update the contradictory status notes before approval.

Review effort: Lite
Findings: 1 Low severity

Open (1)
Previously missed (1)

In code that hasn't changed since last review

Low severity Update authoritative status notes for completed manual assignment

docs/​experiments/​review-protocol.md:22

This now records the manual assignment as complete, but the repository still has two authoritative status notes (docs/implementation/read-only-review.md:25 and docs/experiments/review-summary-edge-cases.md:25) saying that assignment is pending. Please update those references in the same change so readers do not get contradictory gate status.

Copilot AI review requested due to automatic review settings September 24, 2026 01:03

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The experiment setup contains durable protocol and attribution records that warrant final human review.

Review effort: Lite
Findings: None

Resolved since last review (1)

Copilot AI review requested due to automatic review settings September 24, 2026 01:26

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Record full pair-1 SHAs and clarify the remaining-pairs status.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 Medium severity · 1 Low severity

Open (2)

Comment thread docs/experiments/review-protocol.md Outdated
Comment thread docs/implementation/read-only-review.md Outdated
Copilot AI review requested due to automatic review settings September 24, 2026 01:31

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Two documentation consistency and auditability nits remain unresolved.

Review effort: Lite
Findings: None

Resolved since last review (2)

Copilot AI review requested due to automatic review settings September 24, 2026 01:53

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Reconcile stale pair-status wording and contradictory manual-assignment gate documentation.

Review effort: Lite
Findings: None

Copilot AI review requested due to automatic review settings September 24, 2026 02:08
Copilot AI review requested due to automatic review settings September 24, 2026 03:02

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Two unresolved moderate findings require revision alignment before approval.

Review effort: Lite
Findings: 1 Medium severity

Open (1)

Copilot AI review requested due to automatic review settings September 24, 2026 03:24

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Reconcile the protocol table with the description about which trials are prepared before approval.

Review effort: Lite
Findings: 1 Medium severity

Open (1)

Copilot AI review requested due to automatic review settings September 24, 2026 04:04

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Resolve the contradiction between the PR description and the recorded experiment revisions.

Review effort: Lite
Findings: None

Resolved since last review (1)

Copilot AI review requested due to automatic review settings September 24, 2026 04:06

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The experimental protocol and human gate require final human review.

Review effort: Lite
Findings: None

Resolved since last review (1)

Copilot AI review requested due to automatic review settings September 24, 2026 05:26

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

A contradictory prerequisite status remains, and the human review gate is still pending.

Review effort: Lite
Findings: None

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Update the remaining contradictory assignment status in docs/implementation/build-step-1.md.

Review effort: Lite
Findings: None

Copilot AI review requested due to automatic review settings September 24, 2026 05:33

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The protocol and evidence documentation are complex, with two consistency and provenance nits remaining for human review.

Review effort: Lite
Findings: None

@mchwang

mchwang commented Sep 24, 2026

Copy link
Copy Markdown
Contributor Author

Experiment cancelled by product decision before timed results were recorded. This protocol PR will not be merged. Cleanup and roadmap unblocking are in #20; optional future validation is tracked in #19.

@mchwang mchwang closed this Sep 24, 2026
@mchwang
mchwang deleted the codex/freeze-review-experiment-pairs branch September 24, 2026 05:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants